Papers with human evaluation is time-consuming
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023.emnlp-main)
Copied to clipboard
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi
| Challenge: | Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate and (2) human evaluation is time-consuming and costly. |
| Approach: | They introduce a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atom facts supported by a reliable knowledge source. |
| Outcome: | The proposed model breaks a generation into atomic facts and computes the percentage of atomic fact supported by a reliable knowledge source. |